Storing the State in a Remote Backend

Creating a Google storage bucket#

Terraform maintains its internal information about the current state. That allows it to deduce what needs to be done and to converge the actual into the desired state defined in *.tf files. Currently, that state is stored locally in the terraform.tfstate file. For now, there shouldn’t be anything exciting in it. We can see this file in the code playground below by running the cat terraform.tfstate. But for reference, here are the contents of terraform.tfstate.

The field that really matters is resources. It’s empty because we didn’t define any. We’ll do that soon, but we’re not going to create anything related to our GKE cluster. At least not right away. What we need right now is a storage bucket.

Keeping Terraform’s state local is a bad idea. If it’s on a laptop, we can’t allow others to modify the state of our resources. We’d need to send them the terraform.tfstate file through email, keep it on a network drive, or implement some other similar solution. That is impractical.

We might be tempted to store it in Git, but that wouldn’t be secure. Instead, we’ll tell Terraform to keep the state in a Google bucket. Since we’re trying to define infrastructure as code, we won’t do that by executing a shell command, nor will we go to the GCP console. We’ll tell Terraform to create the bucket. It will be the first resource Terraform manages.

But, before we proceed, we need to confirm that billing for storage is enabled. Otherwise, Google won’t allow us to create one.

Note: Follow the instructions in this lesson to fulfill the billing requirement.

Viewing storage.tf#

We’re about to explore the google_storage_bucket module. It allows us to manage Google Cloud storage buckets. More information can be found in the google_storage_bucket documentation.

Let’s take a look at the definition.

• We’re defining storage bucket referenced as state. All resource entries are followed with a type (e.g., google_storage_bucket) and a reference (e.g., state). We’ll see the usage of a reference later in one of the upcoming definitions.

• Just like with the provider, the resource has several fields. Some are mandatory, while others are optional and often have predefined values.

• We’re defining the name and the location. Later on, we specified that it should be created inside our project.

• Finally, we selected NEARLINE as the storage class. Please visit the Available storage classes section of the documentation to see the full list.

Just like before, the values of some of those fields are defined as variables. Others (those less likely to change) are hard-coded.

The labels are there to provide our team members or other people in the organization metadata about our cluster. If we’ve forgotten about it for months, it’s easy to tell who to bother. We also state that we manage the cluster with Terraform, which will hopefully prevent people from making manual changes through the UI.

There’s one tiny problem we need to fix before we proceed.

Google Cloud bucket names need to be globally unique. There can’t be two buckets with the same name anywhere in Google Cloud, among all its users. We’ll generate a unique name using date.

The name of the bucket is now, more or less, unique, and we won’t face the danger that someone else has already claimed it. The environment variable we created will be used as the name of the bucket.

The name of the bucket is also now, more or less, unique, and we won’t face the danger of someone else claiming it. The environment variable we created will be used as the name of the bucket.

Let’s apply the new definition.

The output, limited to the relevant parts, is as follows.

In this case, we can see the full list of all the resources that will be created. The “+” sign indicates that something will be created. Under different conditions, we could also observe those that would be modified (“∼”) or destroyed ("-").

Applying Terraform#

Here Terraform will deduce that the actual state is missing the google_storage_bucket resource. It also shows us what properties will be used to create that resource. We defined some, while others will become known after we apply that definition.

Finally, we’re asked whether we want to perform these actions. We should type “Yes” and press the “Enter” key.

Note: From here on out, we won’t explicitly explain that we need to confirm Terraform actions.

After we choose to proceed, the relevant parts of the output should be as follows.

We can see that one resource was added and that nothing was changed or destroyed.

Since this is the first time we’ve created a resource with Terraform, it’s reasonable to be skeptical that everything worked perfectly. So, we’ll confirm that the bucket was indeed created by listing all of those available in the project. Over time, we’ll gain confidence in Terraform and won’t have to validate that everything works correctly.

Let’s imagine that someone else executed terraform apply and that we’re not sure what the state of the resources is. In such a situation, we can consult Terraform by asking it to show us the state.

The output is as follows.

There’s not much to look at. For now, we have only one resource (google_storage_bucket). As we keep progressing, that output will increase and, more importantly, it will always reflect the state of the resources managed by Terraform.

The previous output is a human-readable format of the state currently stored in terraform.tfstate. We can also inspect that file by using cat terraform.tfstate in the code playground below.

The output is as follows.

If we ignore the fields that are currently empty, and the few that are for Terraform’s internal usage, we can see that the state stored in that file contains the same information as what we saw through terraform show. The only important difference is that one is in Terraform’s internal format (terraform.tfstate), while the other (terraform show) is meant to be readable by humans.

Even though it’s not the case right now, the state could easily contain confidential information. It’s currently stored locally, and we’ve already decided to move it to Google Cloud bucket. That way, we’ll be able to share it, it will be stored in a more reliable location, and it will be more secure.

Moving the state to the bucket#

To move the state to the bucket, we’ll create a gcs (Google Cloud Storage) backend. We’ve already prepared a file backend.tf just for that. The definition of backend.tf is shown below. To see the file in the code playground below, use catbackend.tf.

Viewing backend.tf#

There’s nothing special in that definition. We’re setting the name of the bucket, the prefix, which will be appended to the files, and the path to the credentials file.

The bucket entry in that Terraform definition cannot be set to a value of a variable. It needs to be hard-coded. So, we’ll need to replace devops-catalog with the bucket name we used when we created it.

Let’s apply the definitions and see what we’ll get.

The output, limited to the relevant parts, is as follows.

Since we’re changing the location where Terraform should store the state, we have to initialize the project again. The last time we did that, it was because a plugin (Google) was missing. This time it’s because the init process will copy the state from the local file to the newly created bucket.

The output, limited to the relevant parts, is as follows.

Confirm copying the state by typing “Yes” and pressing the “Enter” key.

The process continues. It copies the state to the remote storage, which, from now on, will be used instead of the local file. Now we should be able to apply the definitions. The output is as follows.

As we can see, there was no need to apply the definitions. The latest addition does not define any new resources. We only added the location for the Terraform state. That change is internal, and it was applied through the init process.

Try it yourself#

You can try all of the commands used in this lesson in the code playground below. Press the run button and wait for a few seconds for it to connect.

Before running the commands used in this section, you first need to connect to GCloud to create the secret keys for the service account. For ease of use, all of the commands above are combined in main.sh.